Prioritizing quality improvements to source code

ABSTRACT

An exemplary method includes receiving source code having a plurality of code segments, providing a desired level of quality for the source code, analyzing the source code to assign a complexity measure to each of the plurality of code segments and assigning a level of code coverage to each of the plurality of code segments based at least in part on the desired level of quality and the complexity measures. An exemplary system prioritizes quality improvements to source code based, in part, on a quality assessment. Such a system can improve code quality by assigning higher test coverage levels to modules with higher complexity.

BACKGROUND

Software developers use various mechanisms to enhance the quality ofsource code during software development. Such mechanisms can addtremendous value by improving quality and user experience as well as byreducing the need for post-release patches. However, such mechanisms canbe costly and constrained.

Developers often aim to reach a certain code quality. Source codequality can be measured on attributes such as performance, compliancewith expected functionality, robustness, complexity, efficiency,portability, usability of the source code, etc. In general, failuresthat occur during testing or during use of a software application serveas indicators of how well the application complies with its expectedfunctionality (i.e., does the application function correctly or not?).

A particular approach to finding failures in a software applicationincludes generation of so-called “test cases”. Once generated, a set oftest cases can be executed to test sections of source code forcompliance with expected functionality. A common definition for a testcase is a set of conditions or variables under which a tester willdetermine if a requirement or use case upon an application is partiallyor fully satisfied. Sometimes it takes many test cases to determine thata requirement is fully satisfied. Thus, a test case can be characterizedby a set of conditions which help to determine whether a requirementupon an application is satisfied or not.

In a test case approach to quality, test prioritization helps todetermine a test execution order or an order of areas for test casedevelopment. For example, some tests may be more pervasive than othersand hence results from a pervasive test may be dispositive, i.e.,eliminate the need for one or more “lower priority” tests. In the latterinstance, test prioritization may simply decide to develop test casesfor highly traveled paths. While no particular logic exists as astandard for test prioritization, test prioritization nevertheless canlower costs by raising quality confidence in an efficient way.

Once some source code is tested (or marked for testing), a code coveragemetric can be assigned. Often code coverage is assigned after executionof a test case, especially where the degree to which the source code ofa program will be tested by a test case is not known a priori.

For all but the simplest of source code, testing is an iterativeprocedure. To assist in this iterative process, software developers usecode coverage tools, for example, to highlight sections of the sourcecode that have not been executed during testing. Risk of failure can bedifficult to ascertain for an untested section. For such sections of thesource code, further test case development can uncover issues and/orverify performance and, hence, improve quality.

As described herein, conventional code coverage tools that merely allowa tester to identify untested code do not add significantly to qualitycontrol. Further, a need exists for test prioritization techniques thatcan make testing more efficient.

SUMMARY

This summary is provided to introduce simplified concepts ofprioritizing quality improvements to source code, which is furtherdescribed below in the Detailed Description. Implementations forprioritizing quality improvements to the source code are also described.

In one implementation, a system for prioritizing quality improvements tosource code receives a plurality of modules associated with the sourcecode as input. The system also receives a desired quality level for thesource code as input. The system determines a complexity measure foreach of the plurality of modules. The system then assigns a testcoverage level to each of the plurality of modules based on thecorresponding complexity measures and the desired quality level for thesource code. The assignment of the test coverage level to each moduleoptionally includes assigning higher test coverage levels to moduleswith higher complexity measures.

This summary is provided to introduce a selection of concepts in asimplified form to be further described below in the DetailedDescription. This summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is set forth with reference to the accompanyingfigures. In general, same reference numbers in different figures referto similar or identical items.

FIG. 1 shows exemplary plots illustrating relationships betweenattributes of bugs present in source code and multiple stages of asoftware application life cycle with respect to time.

FIG. 2 illustrates an exemplary analysis presenting relationshipsbetween code coverage and structural complexity.

FIG. 3 is an exemplary plot of correlation co-efficient versus codecoverage (e.g., branch coverage).

FIG. 4 illustrates an exemplary environment suitable for implementingprioritization of quality improvements to a source code.

FIG. 5 illustrates an exemplary computing device for prioritizingquality improvements to the source code.

FIG. 6 presents an exemplary schematic showing various components of thesystem used to assess a source code's likelihood-to-fail.

FIG. 7 illustrates an exemplary presentation of a per-binary breakdownof the structural complexity and the code complexity information for amanaged binary.

FIG. 8 illustrates an exemplary presentation of sections of source codeand associated code coverage measures, structural complexity metrics andrisk levels.

FIG. 9 illustrates exemplary attributes used to recommend test casedevelopment for one or more code segments.

FIG. 10 illustrates exemplary method(s) for executing and analyzing testcases corresponding to the source code.

FIG. 11 illustrates exemplary method(s) for analyzing complexity andrecommending code coverage tests for the source code based on theanalyzed complexity.

FIG. 12 illustrates exemplary method(s) for implementing prioritizationof test case development

FIG. 13 illustrates exemplary method(s) for identification of locationsin the source code of the software application for test casedevelopment.

FIG. 14 illustrates exemplary method(s) to implement a feedbackmechanism for recursively prioritizing quality improvements to thesource code based on the changes made to the source code.

FIG. 15 illustrates an exemplary computing device capable ofimplementing various exemplary methods described herein.

DETAILED DESCRIPTION

Various exemplary techniques can identify and prioritize test cases toimprove quality of source code. Various techniques involve prioritizingtest cases by assessing a likelihood of source code to fail. Such anassessment can be based on information from already executed test casesor from complexity characteristics and an understanding of relationshipsbetween code coverage, code complexity and code quality.

As explained in more detail below, for some software applications, codecoverage alone, can be a poor or misleading indicator of code quality(i.e., code with higher code coverage can actually have a higher chanceof failure, at statistically significant levels, than code with lowercode coverage).

One measure of quality for a software application is the number ofreported failures. Factors affecting cost to fix “buggy” code includethe number of reported failures for a software application and the stagein the software development cycle at which the reported failuresoccurred. In general, failures found late in the software developmentcycle are more costly to fix. Thus, a strong incentive exists to fixfailures early and to reduce the number of failures that occurring atlater stages.

Source code quality can be improved during software development usingtest cases that test functionality of a software application andidentify failures in earliest of development stages. Another type ofmetric that can improve testing is code complexity. A variety of codecomplexity metrics exist that can identify segments of the source codethat warrant testing.

In one implementation, an exemplary system receives a quality level forsome target source code, for example, as input by a tester. The qualitylevel can be considered a threshold value that indicates a desired levelof quality for the source code. For example, a six sigma qualityprocesses exhibit failures of less than 3.4 per (one) millionopportunities (DPMO). For complex code, given a stated level of quality,testing may not always provide an absolute guarantee; yet, given thestated quality level, testing can provide some assurance (e.g., at someconfidence level). In this implementation, to determine code coveragefor the given quality level, the system relies on code complexity and apriori knowledge of relationships between code complexity, code coverageand code quality. For example, if testing or actual use of a priorversion of an application provided data as to coverage and quality, thecomplexity of the source code for the prior version can be used tounderstand the relationship between complexity, coverage and quality.This relationship can then be used to estimate code coverage for the newversion. Again, as described herein, data indicate that code coveragealone is not always a predictor of quality. In general, the estimatedcode coverage is simply a starting point for an iterative testingprocess where test data are used as feedback for identifying segments ofcode that may require additional coverage to reach a desired level ofcode quality.

In the foregoing implementation, the system computes a complexitymeasure for each of a plurality of code segments. The system can thenassign a test coverage level to each of the code segments based on thecomplexity measure and the desired quality level for the source code.For example, based on some a priori knowledge of a relationship betweencomplexity, coverage and quality, code segments with higher complexitymeasures are assigned higher levels of coverage. Test coverage levelsindicate the number of test cases to be executed for one or more codesegments in an effort to attain a desired quality level. Test coveragelevels are often stated in terms of percentage of source code to becovered by test cases. As described herein, code coverage levels can bedetermined based on attributes such as artifacts from prior releases,commonly used travel path, number of people using the source code,frequency of usage of the code segment, etc.

Multiple and varied implementations and embodiments are described below.In the following section, an exemplary environment that is suitable forpracticing various implementations is described. After discussion of theenvironment, representative implementations of systems, devices, andprocesses for prioritizing quality improvements to the source code aredescribed.

Exemplary Computing Environment

FIG. 1 shows exemplary plots 100 illustrating relationships betweenattributes of bugs present in source code and multiple stages of asoftware application life cycle with respect to time. The plots 100include a plot 102 representing a relationship 104 between cost to fixper bug and the stages of the software development cycle.

The cost to fix per bug is the cost associated with fixing each bug orfailure in a software application. Generally, each bug corresponds to afailure of the software application to comply with an expectedfunctionality. For example, the cost associated with fixing each bug maybe based on the market share lost due to the failure in the softwareapplication. In another example, the cost associated with fixing eachbug may be based on the time required to debug and fix, or rectify thebug. In yet another example, the cost associated with fixing each bugmay be based on the amount of money spent on fixing the bug.

A software development cycle, generally, starts with a developmentstage. This stage includes writing a set of instructions, also referredto as coding hereinafter, which when executed demonstrate the expectedfunctionality of the software application. The development stage isfollowed by a testing stage, where the implemented software applicationis tested for compliance with the expected functionality. A failure inthe software application is reported if the expected functionality isnot met by the application. Generally, a beta release stage occurs aftertesting, which includes exposing the software application to a set ofusers for the purpose of evaluation and getting feedback. The finalstage of the software development cycle is the release or launch stage.In the release stage, the software application is considered to bereasonably “bug free” and market ready as having had most of thefailures fixed.

The relationship 104 represents changes in the cost to fix each bug withrespect to the various stages in the software development cycle. Therelationship 104 shows that the cost to fix each bug increases with eachstage of the software development cycle. In one example, the cost to fixeach bug may be based on the market share lost due to the bug. The costto fix each bug therefore escalates sharply in the launch stage.

In the development stage, the cost to fix each bug is the least as thecoding can be easily corrected as developers often work in coordinateteams that expect bugs to be identified and fixed. The cost to fix eachbug increases during the testing and the beta release stages. Each bugreported during the testing and the beta release stages requiresdebugging of the coded parts of the software application to fix the bug,which can lead to deferment of the release date of the softwareapplication in the market. However, if the bug is reported after therelease, the cost to fix each bug increases exponentially with each bugreported. Failures in the software application, which correspond to thebugs, can decrease the market share of the software application ascustomers choose competing software applications or seek othersolutions.

The plots 100 further include a plot 106 representing the relationship108 between the number of bugs found and the stages of the softwaredevelopment cycle. The relationship 108 represents changes in the numberof bugs found as the stages of software application life cycle progresswith time.

The relationship 108 shows that the number of bugs found in the softwareapplication decreases with each stage of the software development cycle.Generally, during the development stage, the number of bugs found in thesoftware application is the highest. The relationship 108 furtherindicates that the number of bugs found decreases during the testing andthe beta release stages.

Typically, as most of the software applications are executed during thedevelopment stage, a large section of the expected functionality isvalidated in that stage itself, thereby reducing the number of bugsfound in the testing and the beta release stages. The number of bugsfound further decreases in the testing and beta release stages, and istherefore the least when the software application is released in themarket. However, as shown by the relationship 108, the number of bugs inthe software application remain non-zero throughout the release stage.Such dormant bugs may be found and reported even long after the releaseof the software application, especially because users may use anapplication in a manner that was not predicted (e.g., importing fileswith new types of formats, etc.)

To improve the quality of the code and minimize the number of bugs foundafter the testing stage, test cases are developed and executed to coversome percentage of the source code. However, as mentioned, the extent ofcode covered during testing alone is not always a reliable indicator ofcode quality. This point is discussed below with reference to FIGS. 2and 3.

FIG. 2 illustrates an exemplary analysis 200 that presents relationshipsbetween code coverage and code complexity. Code coverage is a measureused to describe the degree to which source code of a software programhas been tested. Generally, a code coverage value (e.g., percent) can becalculated based on test results obtained from executed test cases forsource code of an application. The test cases typically include a set ofconditions to determine whether a requirement upon the softwareapplication is satisfied. The percentage of source code executed by thetest cases can indicate the sections of the source code, which arecovered by the test cases and can thus help evaluate code coverage.

Code complexity, also referred to as the structural complexity, isrelated to the amount of effort needed to understand, maintain, andmodify and the source code correctly. In one implementation, structuralcomplexity is based on the number of linearly independent paths in thecode. For example, structural complexity can be based on the number ofdecision points in the source code. In one embodiment, cyclomaticcomplexity can be used to determine structural complexity of sourcecode. Cyclomatic complexity can be calculated by counting the number ofdecision points, such as if, else, do, while, throw, catch, etc., incode.

The exemplary analysis 200 includes a graphical representation of a lowcomplexity code 202 and a high complexity code 204. The low complexitycode 202 may signify, for example, lower branching within the sourcecode which makes the source code easier to understand and test. The lowcomplexity code 202 may also signify that modifications can be made tothe source code at a lower risk of failure, which can result in lowernumber of bugs for the software application.

The high complexity code 204 may signify, for example, higher branchingcode, which is comparatively difficult to understand and test. The highcomplexity code 204 can therefore warrant more test cases for bettercode coverage.

The low complexity code 202 can include one or more nodes 206-1, 206-2,. . . , 206-N, collectively referred to as 206 hereinafter. In oneimplementation, the nodes 206 correspond to modules in the source code.In another implementation, the nodes 206 can also correspond to aspecific function or a code segment to implement a specificfunctionality in the source code. For example, the node 206-1 mayimplement a functionality of the software application for which the node206-1 is dependent on the node 206-2 for execution. After the control istransferred to the node 206-2, the node 206-2 is further dependent onthe node 206-4 for its execution.

Similarly, the high complexity code 204 can include one or more nodes208-1, 208-2, . . . , 208-N, collectively referred to as 208hereinafter. In one implementation, the nodes 208 correspond todifferent modules, each of which implements a functionality of thesource code. For example, the node 208-1 is a module which implements afunctionality of the software application. The node 208-1 is dependenton the node 208-2 for its execution and, therefore, transfers thecontrol to the node 208-2. Similarly, the nodes 208-4, 208-5, 208-6 aredependent on the node 208-8 for execution, which is in turn dependent onthe node 208-3, making the source code complex to test and understand.

The exemplary analysis 200 presents a plot 210 showing a relationshipbetween the number of test cases (y-axis) with reference to the codecoverage (x-axis) for both the low complexity code 202 and the highcomplexity code 204. Here, 50% code coverage implies that 50% of thesource code is covered by the test cases (i.e., 50% of possiblepathways, etc., have been traversed). This also implies that the rest50% of the source code is uncovered.

The plot 210 shows that for a given code, the number of test casesexecuted is directly proportional to the code coverage. For example, thenumber of test cases executed increases with the increase in the codecoverage percentage for code 202 and also for code 204. For example, thenumber of test cases for low complexity code 202 increases when the codecoverage increases from 50% to 80%.

The plot 210 also shows that the number of test cases for differentsource codes can be different at the same code coverage levels.Conversely, if two source codes are tested using similar number of testcases, the code coverage for the two codes can be different. Forexample, the number of test cases executed for low complexity code 202at 50% code coverage is less than the number of test cases for the highcomplexity code 204 at 50% code coverage while the number of test casesfor the low complexity code 202 at 80% code coverage and for the highcomplexity code 204 at 50% code coverage are similar.

The plot 210 demonstrates that every source code having the same codecoverage percentage may not be interpreted as having been testedequally. Therefore, achieving a threshold percentage of code coveragemay not be a reliable indicator of the code having been adequatelytested to meet a certain quality, especially for higher complexitycodes. The high complexity code 204, for example, being more complex,can be harder to test and thus may warrant higher code coverage. Thisshows that the number of test cases to be executed for the source codeto reach a particular code quality is not only based on a desired levelof code coverage of the source code, but also on the structuralcomplexity of the code. In some cases, if a relationship that linksquality to code coverage alone is used for test case development, thetested code may actually have quality that is quite below the desiredquality.

FIG. 3 shows an exemplary plot 302 of correlation coefficient versusbranch coverage. A code coverage measure may be based on a branchcoverage measure that indicates the percentage of branches in the sourcecode that have been executed. An increase in the branch coverage measurecan signify more instructions being executed and tested. The plot 302can be used to represent a relationship between correlation coefficientof failures in a software application and arc code coverage. Arc codecoverage often refers to a measure of code coverage for branchingstatements such as if, else, while, etc. in source code.

The correlation coefficient is a measure of the correlation betweennumber of failures in a software application and the code coveragemeasures. The correlation coefficient measure can thereby be used toexamine a software application to determine whether code coverage canact as an indicator of number of failures of the application. Generally,it is expected that a higher code coverage results in a lower number offailures, i.e., the correlation coefficient should decrease with anincrease in code coverage. However, in some cases, it can be seen thatthe number of failures uncovered during testing increases withincreasing code coverage, at statistically significant levels.

Plot 302 displayed in the FIG. 3 shows changes in the correlationcoefficient (y-axis) with respect to changes in the branch coveragemeasure (x-axis). The plot 302 shows an increase in the values ofcorrelation coefficients with an increase in branch coverage measureswhen the branch coverage increases over a threshold (e.g., somepercentage). For example, the bar 304 shows the value of the correlationcoefficient as 0.15 when the block coverage is Y %, which increases to0.24 when the coverage increases to a higher percentage Z %. Forexample, for some software applications, the values of the correlationcoefficients decrease when the percentage of code coverage is less than70% and then increase as the percentage of code coverage increasesbeyond 70%. This shows that, in some instances, there is little qualityassurance and high uncertainty, in terms of failures, for a softwareapplication that has a code coverage of less than some particularpercentage. Further, for code coverage greater than this percentage,such an application can have a positive correlation between codecoverage and the number of failures. Therefore, as described herein,code coverage alone is not always a reliable indicator of quality ofsource code for a software application. In other words, for most complexapplications, a quality based on code coverage alone has littleassurance. However, other measures such as complexity of code can beused to improve quality of the code as discussed below.

As described herein, code coverage can be based on any of variety ofmeasures. For example, if a code segment has 30 possible states orpaths, then code coverage may be given as a percentage of states orpaths tested to total states or paths. After tests are run, codecoverage is a short-hand phrase for code execution data, which may be areport or other presentation to comprehend what occurred duringexecution of the tests.

FIG. 4 shows an exemplary network environment 400 suitable forimplementing various techniques for prioritization of qualityimprovements to source code. For discussion purposes, the networkenvironment 400 includes a developer 402, a tester 404, and a supervisor406.

The developer 402 implements functionality of a software application.The developer 402 writes a set of instructions, act of which is referredto as coding, which when executed implement a specific functionality.Depending on specifics of a development cycle, the developer 402 mayalso write and execute test cases for testing the functionality of thesoftware application.

In the environment 400, the tester 404 tests a software application forits compliance with an expected functionality. Again, depending onspecifics of a development cycle, the tester 404 may develop test casesto test the software application. In the environment 400, the testerexecutes test cases, regardless of their origin. The tester 404 reportsfailures (e.g., “bugs”) in the software application after execution ofthe test cases. The test cases are generally classified into white boxtest cases and black box test cases. The white box test cases includetests cases that are written for a software application with knowledgeof the internal working of the source code. The black box test cases, onthe other hand, use a functional perspective of a software applicationto derive a set of test cases. Accordingly, the code coverage of thesoftware application can be calculated based on the extent to which thewhite box and black box test cases test the source code. In theenvironment 400, the supervisor 406 monitors source code and test caseresults to understand better quality of the developer's code.

The developer 402, the tester 404, and the supervisor 406, perform theirrespective tasks and communicate with each other through one or moredevices 408-1, 408-2, . . . , 408-n, collectively referred to as 408hereinafter, through a network 410.

The devices 408 can be implemented as a variety of conventionalcomputing devices, including, for example, a server, a desktop PC, anotebook or portable computer, a workstation, a mainframe computer, amobile computing device, an Internet appliance, a network router, etc.or a combination thereof that are configurable to prioritize qualityimprovements to the source code.

The network 410 can be a wireless or a wired network, or a combinationthereof. The network 410 can be a collection of individual networks,interconnected with each other and functioning as a single large network(e.g., the Internet or an intranet). Examples of such individualnetworks include, but are not limited to, Local Area Networks (LANs),Wide Area Networks (WANs), and Metropolitan Area Networks (MANs).Further, the individual networks may be wireless or wired networks, or acombination thereof.

In one embodiment, a device 408 includes a processor 412 coupled to amemory 414. The memory 414 includes a code coverage module 416, astructural complexity module 418, and a quality assessment module 420.The memory 414 may be coupled to or associated with, and/or accessibleto other devices, such as network servers, router, and/or other devices408.

The code coverage module 416 determines code coverage for the sourcecode. As already explained, code coverage is a measure used in softwareapplication testing to determine the extent to which a softwareapplication has been tested. The code coverage module 416 can alsoanalyze sections of the source code that have not been executed duringthe execution of test cases for the source code to help recommend testcase development. Such recommendations may be performed by the codecoverage module 416 or another module that receives information from thecode coverage module 416. The code coverage module 416 may also estimatecode coverage based on knowledge of similar code (e.g., a prior build orversion or code with similar functionality and complexity) that has beentested.

In one implementation, the code coverage module 416 calculates the codecoverage for a binary of a software application. A binary of a softwareapplication includes source code representation in a processorunderstandable form, which can be executed directly by a processor. Inanother implementation, the code coverage module 416 calculates codecoverage for an intermediate language (IL) code. An IL code is a commonintermediate representation language of a source code, such as MicrosoftIntermediate Language (MSIL), which is generated on compilation of thesource code. MSIL is associated with the .NET framework, which acceptssource code programmed in any of a variety of languages (e.g., objectoriented programming languages such as C++, C#, etc.), which are thentransformed into MSIL to target a common language runtime (CLR).

The structural complexity module 418 determines a complexity measure forsource code that can be used to help assess quality of the code. In oneimplementation, the complexity measure may be used to determine a levelof code coverage that may be required for a desired level of quality.For example, a high complexity measure can indicate higher code coveragerequirement for the source code. Test cases can then be developed (e.g.,by the developer 402 or the tester 404) to attain a pre-determined levelof code coverage and thereby assure better quality compliance of thecode. In one embodiment, the structural complexity module 418 can alsodetermine complexity measures for each code segment of a plurality ofcode segments of the source code. For example, an application may have10 segments where the complexity module 418 assigns one or morecomplexity measures to each segment. In general, each segment provides acorresponding functionality, which may rely on functionality of one ormore other segments.

In one embodiment, the structural complexity module 418 can compute acyclomatic complexity measure to determine the structural complexity ofthe source code. As already mentioned, a cyclomatic complexity measuredepends on number of linearly independent paths through a source code.Thus, an overall complexity measure can be stated for a source code and,optionally, individual complexity measures for segments of the sourcecode.

In the example of FIG. 4, the quality assessment module 420 canrecommend test case development for the source code based on one or morecomplexity measures and a desired level of quality for the source code.In one implementation, the quality assessment module 420 receives aquality assessment value that corresponds to a desired level of qualityfor the source code and also receives one or more structural complexitymeasures for the source code. The quality assessment module 420 can thencompute code coverage required to attain the desired level of quality.The quality assessment module 420 may also provide some recommendationsas to test case development. For example, given a complexity measure ofsource code and a desired level of quality for the code, a level of codecoverage can be determined. In turn, the level of code coverage can beused to estimate how many test cases are required for the code andoptionally for specific segments of the code. For example, code with ahigh complexity measure can lead to a high level (e.g., percentage) ofcode coverage that would thereby warrant more test cases for improvingthe code coverage. Again, as explained with respect to FIG. 3, such atechnique may rely on an a priori knowledge of a relationship betweencode coverage and quality.

In another implementation, the quality assessment module 420 computesthe quality assessment value based on the complexity measure andexisting code coverage. As mentioned, code testing usually occurs in aniterative manner. Hence, after a first iteration, then such an exemplarytechnique may provide information germane to a subsequent iteration.After each iteration, an assessment may be made as to the nature of thecode, for example, is the pattern aligning with a prior build orversion? Or, is it deviating? Where the pattern deviates, then theguidance provided by any preexisting relationship to help determinecoverage with respect to quality may be diminished.

In general, the quality assessment module 420 can weight thelikelihood-to-fail of various code segments, for example, based on oneor more analyses or measures. In turn, the quality assessment module 420can generate a prioritized list of areas to develop tests against.

After a first iteration, a quality assessment value (e.g., failures pertest, etc.) can help in identifying segments of the source code thatwarrant test case development. In such an example, the identificationmay be based in part on some estimated levels of code coverage to reacha certain quality.

As testing usually occurs based on a set of tests, information as toorder of the individual tests can be useful. In general, a test that islikely to expose extremely useful information should be performedearlier than a test that is less likely to expose useful information.For example, a test case that tests a key function of an application canbe positioned earlier in a set of test cases than a test case that testssome less important function, especially where the less importantfunction may be less complex and easier to fix if a failure occurs.Hence, as described herein, an exemplary method can prioritize testcases or prioritize segments of code for testing. In the example of FIG.4, the quality assessment module 420 may prioritize identified codesegments, for example, based on levels of code coverage for the codesegments.

In yet another implementation, the quality assessment module 420 canrecommend test case development based on information, such as artifactsfrom previous releases, usage of the source code, travel path used inthe source code, runtime contexts, manual estimation, etc.

As described herein, a runtime collection mechanism can track data(e.g., values for arguments, globals, etc.), frequently executed codepaths, information regarding thread execution, etc., and use suchinformation to prioritize test cases or test case development. Inparticular, the quality assessment module 420 can rely on structuralcomplexity and/or other information. For example, the module 420 may useone or more analysis measures such as semantic metrics, dependencyanalysis, etc.

Various exemplary methods are described below that may be implementedusing the device 408.

FIG. 5 illustrates various components of the exemplary device 408. Inone embodiment, the device 408 can include, but is not limited to, theprocessor 412, a network interface 502, the system memory 414, and anInput/Output Interface 504.

The network interface 502 can enable the device 408 to receive asoftware application's source code and share results of qualityassessment over a network (e.g., the network 410). For example, thesupervisor 406 of FIG. 4 may monitor changes to the source code of theapplication and development of test cases.

The memory 414 includes computer-readable media in the form of volatilememory, such as Random Access Memory (RAM) and/or non-volatile memory,such as Read Only Memory (ROM) or flash RAM. The memory 414 typicallyincludes data and/or program modules for implementing prioritization ofquality improvements to source code that are immediately accessible toand/or presently operated on by the processor 412. In one embodiment,the memory 414 includes the code coverage module 416, the structuralcomplexity module 418, and the quality assessment module 420. The othermodules 508 may include other software that assists in the functioningof the computing device 408, such as an operating system. The programdata 510 may include an IL code 512, source code 514, a qualityassessment value 516, and other data 518 specific to the system or theapplications.

Where some testing has already occurred, the code coverage module 416can measure code coverage for the source code 514 of the softwareapplication based on the lines of source code that have been executed.The module 416 can also signify the degree to which the source code 514has been tested.

In one embodiment, the code coverage module 416 can evaluate block andarc coverage to help determine a code coverage measure. As mentioned,block code coverage includes a set of instructions that have nobranches. Thus, for evaluating block code coverage, the code coveragemodule 416 assumes that if one line of the source code from the set oflines or instructions is executed, all the remaining instructions wouldbe executed. The arc code coverage, on the other hand, evaluatesconditional statements such as if, while, for, while calculating thecode coverage.

In one implementation, where some testing has occurred, the codecoverage module 416 receives information about executed test cases andthen analyzes the source code 514 to determine a code coverage measurefor the source code and/or code coverage measures for segments of thesource code. The code coverage module 416 can also identify and analyzesections of the source code 514, which are not covered by the testcases. A test case development process can then be implemented toincrease the percentage of coverage for the source code. Such a processincludes providing and executing tests to cover the sections of thesource code 514 that have not been tested. Where suitable test cases arenot already written, the process may include writing test cases.

In another implementation, the code coverage module 416 can also analyzea binary or an IL representation of the software application. The binaryrepresentation includes presentation of the software application in aprocessor or a machine readable format. The IL representation is atransformed version of a source code. Object-oriented programminglanguages such as C#, Visual Basic (VB), C++ can all be represented inthe common IL representation such as MSIL.

For MSIL, when compiling to managed code that targets the .NET frameworkCLR, a compiler translates the source code into MSIL, which is aCPU-independent set of instructions that can be efficiently converted tonative code for a particular computing architecture. MSIL includesinstructions for loading, storing, initializing, and calling methods onobjects, as well as instructions for arithmetic and logical operations,control flow, direct memory access, exception handling, and otheroperations. Before MSIL code can be run on a computing device, it mustbe converted to CPU-specific code, usually by a just-in-time (JIT)compiler. Because the .NET framework CLR supplies one or more JITcompilers for each supported computer architecture, the same MSIL codecan be JIT-compiled and run on any supported architecture.

In yet another implementation, the code coverage module 416 can alsoanalyze specific code segments of the source code. The code coverage foreach code segment can be computed independently based on sections of thecode segment that are not executed by the test cases also referred to asthe uncovered sections. The code coverage module 416 can also analyzedistributed applications (e.g., Web 2.0 applications) to calculate codecoverage.

In one implementation, the code coverage module 416 determinespercentage of uncovered source code. The uncovered source code includessections of the source code that have not been executed by the testcases. For example, 95% uncovered code coverage implies that only 5% ofthe entire source code has test cases corresponding to it to test thefunctionality. The code coverage measure may be used in addition to thestructural complexity measure to assess quality compliance of the code.

The structural complexity module 418 determines one or more complexitymeasures for the source code 514. In one implementation, a complexitymeasure is used to determine the level of code coverage that may berequired for a desired level of quality of the source code 514. Forexample, a higher complexity measure can indicate higher levels ofrequired code coverage. This indicates that the more complex the sourcecode 514 is, higher is the code coverage requirement, and more thenumber of test cases that need to be written for the desired level ofquality.

In one implementation, the structural complexity module 418 uses acyclomatic complexity measure to calculate complexity of source code.Cyclomatic complexity uses graph theory and is based on the number oflinearly independent paths in a source code. Nodes of the graphcorrespond to instructions of the source code, and the edges correspondto the sequence of execution of the commands.

The structural complexity module 418 can also identify structuralcomplexity metrics such as coupling, and depth of inheritance toevaluate the structural complexity of the source code. The depth ofinheritance is based on the hierarchy of a code segment in the sourcecode. The deeper the code segment is in the hierarchy, the greater thenumber of code segments it is likely to depend on, making it morecomplex to predict its behavior. Coupling, on the other hand, indicatesthe dependency of one code segment over another. A higher couplingbetween two code segments corresponds to a higher structural complexity.

In one embodiment, the structural complexity module 418 measures thestructural complexity of each code segment of the source code thatprovides a specific functionality to the software application. Inanother embodiment, the structural complexity module 418 can alsomeasure the structural complexity based on an intermediaterepresentation 512 of the source code 514 (e.g., IL). In yet anotherembodiment, the structural complexity module 418 calculates thecomplexity of one or more components of the software application. Eachcomponent can be a logical block of the source code for which one ormore complexity metrics can be extracted.

In one implementation, the structural complexity module 418 analyzes thestructural complexity using a binary metadata reader. The binarymetadata reader extracts metrics (e.g., associated with complexity) fromthe binary of the source code 514. In the .NET framework, metadata isbinary information stored in the binary describing the source code 514.When the source code 514 is compiled into the binary, also known as aportable executable (PE) file, metadata is inserted into one portion ofthe binary, while the IL code 512 is inserted into the other portion ofthe binary. The metadata references code segments, types of data, etc.,in the source code 514 so that when the source code 514 is executed, themetadata can be loaded first in the memory to give information about thecode segments, code members, inheritance, and so on.

The quality assessment module 420 uses a quality assessment value torecommend test case development for the source code. In oneimplementation, the quality assessment value may correspond to a desiredlevel of quality for the source code. In one implementation, the desiredlevel of quality can be determined for each code segment based onartifacts from previous releases, usage of the source code, frequency ofusage, travel path used in the source code, runtime contexts, manualestimation, binary owners, etc.

The quality assessment value can then be used to identify the level ofcode coverage required for each code segment, for attainting the desiredlevel of quality based on the complexity measure of the source code.Such level of code coverage can then be used as a reference to write newtest cases to improve the existing code coverage. For example, higherlevels of code coverage can be associated with a source code havinghigher complexity measures when compared with another source code havinglower complexity measures.

In another implementation, the quality assessment module 420 computesthe quality assessment value based on the existing code coverage andcode complexity measure of the source code 514. An exemplary qualityassessment value can be expressed as follows:Quality Assessment Value=Code Complexity/Code Coverage.

A quality assessment value can help identify code segments of the sourcecode 514 that warrant test case development by identifying new levels ofcode coverage for the code segments. The quality assessment module 420can also prioritize the identified code segments based on the qualityassessment value. For example, a source code with lower code coverageand high complexity measure can have higher priority for test casedevelopment as compared to a source code with lower code coverage andlow complexity measure.

In yet another implementation, the quality assessment module 420identifies code segments of the source code 514 that need refactoring.The identification includes finding code segments of the source code514, which are complex and less understandable. Such code segments canalso be identified based on the complexity measures of the source code.The identified sections of the source code 514 can be re-written to makethe sections more testable and simpler to understand.

The level of code coverage can be used to indicate completeness of atest case development process, for example, relative to a desiredstandard of reliability. The code segments can also be prioritized basedon the level of code coverage. For example, code segments with highercomplexity can have higher levels of code coverage and be assigned ahigher priority.

Multiple quality assessment values R1, . . . , Rn, for n different typesof code coverage measures and complexity measures, can be provided. Thecode coverage measures include measures such as block coverage measure,arc coverage measure, statement coverage measure, function coveragemeasure, and branch coverage measures. The complexity measures caninclude complexity metrics such as coupling, depth of inheritance. Forexample, consider the following metrics:R1=Overall complexity (method, file, binary, component)/Block Coverage;andR2=Overall complexity (method, file, binary, component)/Arc Coverage.

In one implementation, the overall complexity measures mentioned aboveincludes complexity at various levels of the source code 514. Forexample, the complexity can be calculated for a specific method of thesource code 514 or for the whole file of the source code 514. Thecomplexity can also be calculated for the binary using the binarymetadata reader, or for a specific component of the source code 514.

In an exemplary method, a quality assessment value is used for settingexpected code coverage bounds in a manner that accounts for complexityof a selected source code. For example, if R1 and R2 measures are set to30.0 and 25.0 respectively, then a method in the source code (e.g., asegment of the source code) with a cyclomatic complexity of 150 shouldhave a code coverage level of 5% and a method with a cyclomaticcomplexity of 1500 should have a code coverage level of 50%. Such anapproach to quality is relative and can be adjusted according to thecomplexity of a selected source code to determine appropriate codecoverage levels between 0-100%.

In the example of FIG. 5, the quality assessment module 420 can alsocompute the level of code coverage based on different weightings andcoefficients associated with different code coverage and structuralcomplexity measures. In this example, the different weightings andcoefficients are adjustable, for example, based on the relevance of thecode coverage and the structural complexity measures. For example, anarc code coverage measure may not be very relevant in a source code thatdoes not have many conditional or branching expressions.

In one implementation, one or more levels of code coverage can becalculated for a list of code segments based on the desired level ofquality and the structural complexity measures. In such animplementation, the quality assessment module 420 prioritizes a list ofmethods for test case development based on the level of code coverage ofeach code segment. For example, a code segment with a structuralcomplexity of 1200 would have a higher priority over a code segment witha structure complexity of 1100.

In another implementation, the quality assessment module 420 calculatesthe quality assessment value corresponding to the desired level ofquality based on a set of artifacts used from one or more previousreleases of a software application. The artifacts may include attributessuch as, bugs reported, people involved in development of the softwareapplication, usage of the software application, etc., that can helpgenerate data to improve quality of the source code 514 of the softwareapplication. The quality assessment module 420 can receive fine tuningdata and calculate a quality assessment value based on the fine tuneddata. For example, the bugs found in a previous release can help toassess common types of failures in a software application and can thushelp adjust one or more coefficients for a selected source code forpurposes of calculating a quality assessment value.

In yet another implementation, the quality assessment module 420calculates the level of code coverage for prioritizing the code segmentsbased on the travel path of the code segment within the source code. Thetravel path can be based on attributes such as context, the frequency ofusage, security concerns, the number of people using it, etc. Forexample, a code segment having the same code coverage and the structuralcomplexity as another section but having a higher travel path can beassigned a higher priority for test case development.

In one embodiment, the quality assessment module 420 can identify codesegments of the source code 514 that warrant further test developmentbased on an object code and a program database (PDB) file correspondingto the source code 514. In this example, the object code, also referredto as the object file, includes a representation of the source code 514that a compiler or assembler generates after processing the source codefile 514. Typically, the PDB file holds debugging and source codeinformation and can help in retrieving source code information from theobject code. Such information can be used for any of a variety ofpurposes for test case development and, optionally, prioritization oftest cases.

Further, the quality assessment module 420 can incorporate a feedback ora learning loop to enhance quality assurance. A feedback or learningloop can include calculating a level of code coverage recursively basedon changes made to the source code 514. For example, consider a scenariowhere the quality assessment module 420 calculates a level of codecoverage for a code segment of the source code 514 that corresponds toone or more methods. The code segment can then be refactored based onthe level of code coverage. The quality assessment module 420 can thencalculate a modified level of code coverage based on the refactoredsegment. Such a feedback mechanism can help evaluate the improvements inthe source code 514. The feedback can also help in getting a newprioritized list of methods every time the source code 514 is changed orrefactored. The quality assessment module 420 can also align parametersand coefficients associated with the code coverage measures and thestructural complexity metrics based on the feedback.

In one implementation, once the quality assessment module 420 hasanalyzed the code, it proceeds to prioritize a list of methods based ona desired level of quality (e.g., for each individual method or thesource code as a whole) and structural complexity measures for each ofthe methods in the list. The quality assessment module 420 can presentthe prioritized list of methods to a user on a user interface. Thepresentation of the prioritized list of methods can help a developer 402(e.g., or a tester 404 or supervisor 406) visualize a method'sreliability by mapping the source code location of the method to one ormore analyses provided by the quality assessment module 420. Eachanalysis can be based on the coefficients and weightings of the codecoverage and structural complexity measures. In one embodiment, thequality assessment module 420 prioritizes the sections of the sourcecode 514 and also presents some lines of instructions for one or more ofthe sections, optionally with a corresponding analysis. Where a sectionof the source code 514 is adequately covered and error free, the sectionmay be marked or listed as needing no further testing.

Further, the quality assessment module 420 can associate risk levels toeach method or section of the source code 514. The risk levels, whichcan be ascertained based in part on a quality assessment value, canindicate a likelihood of failure. For example, the risk levels may bepresented in the form of a tri-state setting indicating a high level ofrisk, a medium level of risk, or a low level of risk. In an embodiment,the quality assessment module 420 can also specify an arbitrary numberof priority levels as numeric indexes. The quality assessment module 420can also alter underlying thresholds, which determine the priority levelof risk to be associated with a method or section of the source code514.

FIG. 6 presents an exemplary schematic 600 showing one or morecomponents for the code coverage module 416, the structural complexitymodule 418, and the quality assessment module 420. The code coveragemodule 416 includes a dynamic analysis component 602 and the structuralcomplexity module 418 includes a static analysis component 604configured to assess a source code's likelihood-to-fail. The dynamicanalysis component 602 analyzes the code coverage of a softwareapplication based on the execution of the software application throughthe test cases. The static analysis component 604 performs analysis ofthe structural complexity of a software application without executingthe software application.

The dynamic analysis component 602, also referred to as runtimecomponent, includes a binary 606 corresponding to a softwareapplication. In one implementation, the dynamic analysis component 602is configured to provide code coverage information of the binary to acoverage store 608. The code coverage information includes the usage ofthe binary in terms of what sections of the source code of the softwareapplication are executed through the test cases. The information canalso include the amount of time for which section runs and its frequencyof execution.

The dynamic analysis component 602 can also be associated with a runtimecontext to evaluate the binary's likelihood-to-fail. The runtime contextcan be based on whether a software application runs on a server or on aclient machine. A software application running on a server machine, forexample, can include high value information (e.g., stock transactions,banking, etc.) and cost of failure may be significant. Softwareapplications that can fail an corrupt data can also introducesignificant cost; such applications may also be classified ashigh-priority software applications. Another attribute of a runtimecontext can be security concerns of a software application.

In the example of FIG. 6, the static analysis component 604 isconfigured to extract complexity metrics from the binary 606. In oneimplementation, the binary 606 is segregated based on classes 610-1, . .. , 610-M, referred to as 610 hereinafter. The complexity metrics 612-1,. . . , 612-M, referred to as 612 hereinafter, can be extracted for eachclass 610. In another implementation, the binary 606 can also beclassified based on code segments of the source code or onmethods/functions of the source code.

The exemplary schematic 600 further includes one or more components forthe quality assessment module 420. The quality assessment module 420 canassess likelihood-to-fail based on the complexity measure and a desiredlevel of quality for the source code. In one implementation, the qualityassessment module 420 determines a level of code coverage required forattaining a desired quality based on the complexity measure and thequality assessment value 516.

The quality assessment module 420 includes a dynamic/static analysiscomponent 614, a runtime context component 616, and a binary owner'scomponent 618 to determine the quality assessment value 516. Thedynamic/static analysis component 614 includes determining the qualityassessment value 516 based on code coverage and one or more complexitymeasures for source code. For example, an existing level of codecoverage can be determined by the code coverage module 416 based on theuncovered percentage of source code. One or more complexity measures canbe computed by the structural complexity module 418 to determine thecomplexity of the code. The quality assessment value 516 can then bedetermined based on the code coverage and complexity measures.

The quality assessment module 420 can determine the quality assessmentvalue 516 based on the runtime context component 616. The runtimecontext component 616 determines the software application'scharacteristics such as runtime setting (e.g., client or server) andsecurity.

The quality assessment module 420 can also determine the qualityassessment value 516 based on the binary owner's component 618. Thebinary owner's component 618 includes a static analysis mechanism todetermine the quality assessment value 516 based on the historicalknowledge of the software application. A binary owner can include asoftware developer that manages and develops the software application.In one embodiment, if 60% of a development team is new and lackshistorical knowledge, the binary owner's component 618 can provide suchinformation (e.g., newness of the team), which, in turn, may be used toassess quality. Experience of an individual developer or a team may beused in conjunction with measures such as structural complexity todetermine a quality assessment value. For example, complex codedeveloped by an inexperienced developer may benefit from more testing(i.e., greater code coverage to reach a desired level of quality).

FIG. 7 shows an exemplary presentation of quality assessment information700 that can be displayed to a user on a graphical user interface (GUI).The presentation 700 shows a binary hierarchy 702 with associatedstructural complexity 704, code coverage information 706 and risk level708. The example of FIG. 7 pertains to managed binaries; as mentioned,the .NET framework provides a CLR for execution of managed binaries.

The hierarchy 702 organizes the source code by namespaces, types, andmembers of the source code. The hierarchy 702 can include an author, adate, and a time attribute 710. The author attribute shows the creatorof the binary, whereas the date and the time attributes show thetimestamp of when the binary was created or modified.

The structural complexity 704 shows the structural complexity measuresfor each class of the binary. For example, the“Microsoft.Expression.Web” class has a structural complexity of “787”.In one implementation, the structural complexity is extracted from an ILrepresentation of the code (e.g., using a cyclomatic complexitymeasure).

In the graphic 700, the code coverage metric 706 indicates percentage ofthe source code that is not covered (i.e., uncovered) by the test cases.In one embodiment, the percentage of blocks that are not covered by thesource code, is an indicator of the sections of the source code that donot have corresponding test cases. For example, the“Microsoft.Expressions.Web.Profiling” class has an uncovered percentagewhich equals to 100%, thereby implying that there are no test casesexecuted for the class. This can also imply that there are no test caseswritten for this class.

As mentioned, the quality assessment module 420 can associate a risklevel 708 with each class of a binary. In one implementation, the risklevel associated with each class is based on the level of code coveragerequired by the class, which in turn is determined based on the qualityassessment value 516 and the complexity measures identified by thestructural complexity module 418. For example, source code with highcomplexity measures leads to higher level of code coverage requirement.The risk level 708 for a segment of code may change as such code istested and as coverage increases or, alternative, the risk level 708 mayindicate a general level of risk (e.g., associated with complexity,historical information, etc.).

The exemplary presentation 700 associates the risk level 708 in the formof a tri-state setting indicating a high level of risk, a medium levelof risk, or a low level of risk. The risk level column 708 is sortable(as are all others), for example, by clicking on a column heading. Inanother implementation, numeric indexes rather than tri-state icons canbe reported as the risk level 708, which may be sortable and which canprovide more granularity for evaluating results.

FIG. 8 shows an exemplary presentation of quality assessment information800 that can be displayed to a user on a graphical user interface (GUI)(e.g., consider the devices 408 of FIG. 4). The presentation 800displays the structural complexity measures 704, the code coveragemeasures 706 and the risk levels 708 corresponding to various segmentsof source code.

The presentation 800 includes a view 802 showing the hierarchy 702drilled down to the member/function level view and showing data relatedto methods and functions. The view 802 also shows an author and atimestamp 710. Each function in the hierarchy 702 is associated with acorresponding structural complexity measure and a corresponding codecoverage measure. For example, the function ShellExecute (string, bool,class, string) has a structural complexity of 480 and a 100% uncoveredcode implying that no test cases have been executed (and/or written) totest and execute the function.

The presentation 800 also includes an Integrated Development Environment(IDE) view 804 showing source code corresponding to a function 806highlighted in the view 802. In one implementation, a function in theview 802 is selected by double-clicking the function. In anotherimplementation, a function is selected using the toolbar 808 in the view802. The toolbar 808 can also be used to delete a function from the listof functions in the view 802.

The view 804, as described above, shows the source code corresponding tothe function 806 highlighted in the view 802. For example, thehighlighted function 806 refers to the function ShellExecute (string,bool, class, string) having a structural complexity of “480” and a 100%uncovered code. A 100% uncovered code suggests that no test cases havebeen executed (and/or written) for the function, and thus a high risklevel can be associated with the function in view of its structuralcomplexity.

The marked section of the source code 810 in the view 804 indicates thepath coverage of the source code through the function. The markedsection 810 indicates that there has been no coverage for the code inthis section and that the function has not even been entered. The markedsection of the source code 810 can also be highlighted in color to showthe travel path coverage of the source code through the function. Thespecific color associated with unexecuted paths can indicate afunction's priority level like high-priority, medium priority, or lowpriority. An icon 812 on the left margin can also be used to indicatethe priority, such as a high priority, given to the highlighted function806 to focus on for test case development. In the example of FIG. 8,based on a quality assessment value which corresponds to a desired levelof quality and on a complexity measure (e.g., which is very high: “480”in the above example function 806), the selected portion of the functioncan be determined to have a high code coverage level requirement.

The visual presentation 800 can also be changed to a textual view. Thetextual view includes a numeric risk/priority value associated with thefunction rather than a colored indicator of the risk/priority.

The visual presentation 800 can also be based on a binary and a sourcecode view. The binary view can include a set of “.dll” files of thebinary of the software application. The binary view can map classes inthe “.dll” files with the risk levels based on the levels of codecoverage. Locations in the classes can also be identified based on thesections of the classes that warrant further test case development.

The source code view, on the other hand, includes a set of source codefiles such as, .cpp files, within a directory. The locations in thesource code files can then be identified based on the sections of thesource code files that warrant further test development. For example,consider a directory “x” that includes two source code files, “a.cpp”and “b.cpp”. The locations can be identified for the source code filesbased on the risk levels associated with the files, which can beoptionally based on associated levels of code coverage.

The visual presentation 800 can further include a people view 830. Thepeople view 830 can include a list of members (e.g., a set of names suchas, Mike, Jeff, etc.) within a group and associate the differentsegments of the code with the members of the group (e.g., X1-14, X15-73,etc.). The view 830 can also associate risk levels with each personbased on an assessment of the quality of source code written by theperson and, for example, months of experience. For example, consider theGroup X that includes members 1 through 5 where the overall group has anaverage experience of 32 months and a low risk. The risk levels can alsobe associated with each person based on the level of code coverage forthe source codes written by them, which is further based on the desiredlevel of quality and complexity measures of the source code.

In addition to the level of code coverage, other attributes such as,number of users using the code, frequency of usage, context orfunctionality of the code, and security level required can also help indetermining the extent of test case development required as shown inFIG. 9.

FIG. 9 illustrates exemplary schematic 900 including exemplaryattributes 902 that can be used in an analysis 904 to recommend testcase development for one or more code segments 906. Multiple attributessuch as percentage of users using the code segment, frequency of usage,context of the code segment, security level required, and number of bugsfound in similar code segments can help prioritize the levels of codecoverage required to attain the desired level of code quality.

For example, the exemplary parameters 902 include multiple attributesfor one or more code segments. In one embodiment, the code segments aredifferentiated based on the context of the code segment. For example,the top menu code segment provides a navigation bar that helps innavigation of the application. Similarly, the music pages generallyrelate to browsing of the songs.

At step 904, the code segments are analyzed based on the attributes ofthe code segments. The analysis includes comparison of the attributesfor one or more code segments to determine relevance and importance ofthe attributes to the code segment. For example, thepurchasing/transactions code segment requires a high security level andhalf of the users accessing the application also use this code segmentattaching a very high level of importance to the code segment. Videopages, on the other hand, require low security levels and are mainlybrowsed making them relatively lower in priority as far as the test casedevelopment is concerned.

At step 906, test case development recommendations are given based onthe analysis done at step 904. For example, purchasing/transactions codesegment can have a higher level of test case development recommendationbased on nature of the attributes. The test case developmentrecommendations can also be associated with the levels of code coverage.For example, a code segment with a high level of code coverage needshigher number of test cases to improve the quality of the code andthereby is given a higher level of test case development recommendation.

Exemplary Methods

Exemplary methods for implementing prioritization of qualityimprovements to code are described with reference to FIGS. 1-9. Theseexemplary methods can be described in the general context of computerexecutable instructions. Generally, computer executable instructions caninclude routines, programs, objects, components, data structures,procedures, modules, functions, and the like that perform particularfunctions or implement particular abstract data types. The methods canalso be practiced in a distributed computing environment where functionsare performed by remote processing devices that are linked through acommunication network. In a distributed computing environment, computerexecutable instructions may be located both in local and remote computerstorage media, including memory storage devices.

The exemplary methods are illustrated as a collection of blocks in alogical flow graph representing a sequence of operations that can beimplemented in hardware, software, firmware, or a combination thereof.The order in which the methods are described is not intended to beconstrued as a limitation, and any number of the described method blockscan be combined in any order to implement the methods, or alternatemethods. Additionally, individual blocks may be deleted from the methodswithout departing from the spirit and scope of the subject matterdescribed herein. In the context of software, the blocks representcomputer instructions that, when executed by one or more processors,perform the recited operations.

FIG. 10 illustrates an exemplary method 1000 for executing and analyzingthe test cases corresponding to the source code. The method 1000includes iterative development of test cases based on the desired levelof quality. In one implementation, once the desired level of codequality is achieved, the test case development can be stopped and thetest cases can be stored in the test case repository.

In the example of FIG. 10, the method 1000 includes an internal feedbackloop and an external feedback loop. The external feedback loop pertainsto information obtained from field testing such as beta testing whereasthe internal feedback loop pertains to tests performed within the guiseof a controlled testing environment with controlled tests (i.e., testsperformed using proscribed test cases).

The method 1000 may be used for newly developed code 1002 or for fieldimplemented code 1004. In general, data are available for fieldimplemented code 1004. Such data can be used for any of a variety ofpurposes such as quality assessment, test case development, etc.

At block 1006, the newly developed code 1002 or the field implementedcode 1004 are received. As already mentioned, code may include codeinformation such as metadata that gives description about the structureand behavior of code such as class structure, data types of variables.The metadata can also be used to extract structural complexity measuresfor received code.

At block 1008, test cases are determined for the code based on thedesired level of code quality required. For example, levels of codecoverage are determined based on the desired level of quality and on thecomplexity measures for the code. The level of code coverage canindicate the priority and extent for test case development based onwhich the test cases are determined.

At block 1010, the determined test cases are executed for the code(i.e., tests are performed). At block 1012, execution results of thetest cases are analyzed to check whether a desired level of code qualityis achieved. The code coverage module 416 can also analyze the executionresults to compute code coverage for the code. The code coverage canthen be used in the analysis.

At block 1014, a decision occurs as to the quality of the code, forexample, whether the results indicate that failures are below a desiredlevel and that code coverage is above a proscribed level. The decisionblock 1014 may involve comparing the analyzed test results with thedesired level of quality to determine whether additional tests should beperformed.

At block 1016, if the code quality is at or above a desired level ofquality, i.e. the “yes” branch from block 1014, the code is declared tohave passed and no further test cases may be required. In other words,the level of code coverage is deemed acceptable. The process thenproceeds via an optional external feedback loop, which includes a fieldtesting block 1022 (e.g., beta or other field release).

At block 1018, on the other hand, if the results are not “OK” (i.e., notmeeting one or more quality related criteria), i.e. the “no” branch fromblock 1014, then the quality assessment module 420 identifies one ormore code segments that warrant further testing (e.g., test casedevelopment). The identified code segments require further testing toimprove corresponding levels of code coverage to attain the desiredlevel of quality.

At block 1020, additional test cases can be determined for theidentified code segment or segments, for example, based on the desiredlevel of quality and the complexity measures of the code. The additionaltest cases can then be executed at block 1010 forming an internalfeedback loop that aims to improve code quality.

For the external feedback loop, test cases performed and associated testinformation (e.g., including complexity, etc.) can be stored in a testcase repository and can be referred to at block 1006 after field testingof the code.

FIG. 11 illustrates exemplary method(s) 1100 for analyzing complexityand recommending code coverage tests for the source code based on theanalyzed complexity.

At block 1108, an analysis module (e.g., the quality assessment module420) receives source code written in a programming language code 1102(e.g., C#, C++, etc.), in an intermediate language 1104 (e.g., MSIL) orin a very elementary web application language 1106 (e.g., HTML). Theanalysis block 1108 analyzes the received code for complexity.

At block 1110, recommendations as to code coverage for a desired levelof quality are provided. For example, given some a priori knowledge ofcomplexity, code coverage and quality, a module may recommend codecoverage levels as a function of complexity and a desired level ofquality. In one implementation, the quality assessment module 420computes the level of code coverage from a desired level of quality andthe complexity measures. The quality assessment module then recommendsthe code coverage tests accordingly. The code coverage tests indicatethe test cases required for one or more code segments of the code forattaining the desired level of quality.

At block 1112, the code coverage tests are executed for the receivedcode. At block 1114, the executed code coverage tests are analyzed. Suchan analysis may compute the code coverage of the code and estimate thequality of the code.

Overall, the method 1100 may be used as an initial step in testing code,which may be followed by one or more subsequent iterations, for example,based on the assessment of the test results per block 1114.

FIG. 12 illustrates exemplary method(s) 1200 for implementingprioritization of test case development.

At block 1202, a device 408 accepts a list of code segments from aparticular source code. In one implementation, the list of code segmentsmay also be from different source code files instead of being a part ofthe same source code file. In another implementation, the input to thedevice 408 can also be an IL representation of the source code such asMSIL. In yet another implementation, the device 408 can also accept oneor more sections of the source code, where each section represents a setof instructions from the source code.

At block 1204, a desired level of quality for the code segments isprovided, for example, by a tester. At block 1206, structural complexityof the segments are evaluated, for example, to provide complexitymeasures for each code segment of the source code (see, e.g., thestructural complexity module 418). In one implementation, the structuralcomplexity module 418 extracts complexity measures based on thecyclomatic complexity measure. The complexity measures, also referred toas complexity metrics, can include metrics such as coupling and depth ofinheritance.

At block 1208, levels of code coverage, also referred to as coveragelevels, are determined for the code segments based on the complexitymeasures and the desired level of quality, for example, by the qualityassessment module 420.

At block 1210, the list of code segments are prioritized based at leastin part on the levels of code coverage. For example, a code segment withhigh complexity code may have a higher required level of code coverageand thus a higher priority for test case development when compared to acode segment with a low complexity. In one implementation, the qualityassessment module 420 prioritizes the test cases.

At block 1212, the prioritized list of code segments is presented to auser using graphical user interface visualizations. In oneimplementation, the presentation lists all methods along withcorresponding levels of code coverage and corresponding structuralcomplexity measures. Moreover, the risk levels can also be presentedwith each code segment. The presentation can also include an IDE view tomap each code segment of the view directly to the source code of thecode segment.

FIG. 13 illustrates an exemplary method 1300 for identification oflocations in the source code of the software application for test casedevelopment.

At block 1302, the device 408 accepts source code in IL representation(e.g., MSIL). In one implementation, the device 408 can accept thesource code directly in any implementation language such as C#, VB, C++.

At block 1304, the device 408 provides test cases written for the code.At block 1306, the structural complexity module 418 extracts thecomplexity metrics, also referred to as complexity measures, based onthe IL code. In one implementation, the structural complexity module 418extracts the structural complexity metrics, such as coupling, depth ofinheritance, from the IL representation of the source code. In anotherimplementation, the structural complexity module 418 extracts thestructural complexity metrics from the object code and PDB files.

At block 1308, an analysis analyzes the provided written test cases incombination with the structural complexity metrics. At block 1310,locations that warrant further test case development are identified inthe IL representation based on the analysis. In one implementation, thequality assessment module 420 identifies the locations in the sourcecode from the IL representation based on the analysis.

At block 1312, the identified locations are presented to the user. Inone implementation, the quality assessment module 420 presents theidentified locations to the user on a user interface.

In the example of FIG. 13, the method 1300 allows a user to identifyadditional test cases based on existing test cases and complexitymetrics. Such a method may be performed prior to execution of any testcases or may occur after execution of one or more test cases. Whereresults from testing are available, such information may be used to helpdetermine locations in the code for additional test case development.

FIG. 14 illustrates an exemplary method 1400 to implement a feedbackmechanism for recursively prioritizing quality improvements to sourcecode.

At block 1402, a device (e.g., device 408-1, 408-2, 408-3) receivessource code. In one implementation, the device can accept the sourcecode in any implementation language such as C#, VB or C++. In anotherimplementation, the device 408 can accept the source code in an ILrepresentation. In yet another implementation, the device receives codewritten in a mark-up language (e.g., Extensible Markup Language (XML) orHyperText Markup Language (HTML)).

At block 1404, the device computes code coverage measures and structuralcomplexity metrics for the source code. In one implementation, the codecoverage module 416 computes the code coverage based on measures such asstatement coverage, block coverage, and arc coverage. In such animplementation, the structural complexity module 418 can extractcomplexity metrics based on the cyclomatic complexity measure of thesource code.

At block 1406, a set of locations are identified in the source codebased on an analysis of the code coverage, the structural complexitymetrics, and other factors such as the number of people using the sourcecode, the frequency of usage, and the number of bugs reported. In oneimplementation, the quality assessment module 420 identifies thelocations in the code based on sections of the code that are difficultto understand and difficult to test, and thereby warrant refactoring tomake the sections simpler to understand and test (e.g., and debug).

At block 1408, the source code is refactored at the identified sourcecode locations. The refactoring includes changing the source code basedon the analysis of the code coverage, the structural complexity metrics,and other factors such as the number of people using the source code,the frequency of usage, and the number of bugs reported.

At block 1410, modified code coverage and structural complexity metricsare computed based on the refactored source code.

At block 1412, modified locations are identified in the refactoredsource code based on the modified code coverage measures and structuralcomplexity metrics. For example, changes in the source code can reducethe complexity of the lines of instructions at one of the multipleidentified locations. This location may then not be identified in themodified location as it may have low structural complexity metrics thatmay not need further test case development. However, while reducing thecomplexity at this location, the complexity at another location may haveincreased, which may then be reported as the modified location. Thus,changes in the source code can be given as a feedback to the qualityassessment module 420, which can then identify the modified locationsbased on the feedback and the modified code coverage measures, thestructural complexity metrics, and other factors such as the number ofpeople, the usage, the frequency of usage, the number of bugs reported.

At block 1414, the modified locations are presented to the user. Theuser can then again change the refactored code to improve the quality ofthe source code.

Exemplary Computing Device

FIG. 15 illustrates an exemplary computing device 1500 that may be usedto implement various exemplary modules (or components) and in forming anexemplary system. For example, the devices of FIG. 4 may include variousfeatures of the device 1500.

In a very basic configuration, computing device 1500 typically includesat least one processing unit 1502 and system memory 1504. Depending onthe exact configuration and type of computing device, system memory 1504may be volatile (such as RAM), non-volatile (such as ROM, flash memory,etc.) or some combination of the two. System memory 1504 typicallyincludes an operating system 1505, one or more program modules 1506, andmay include program data 1507. The operating system 1505 include acomponent-based framework 1520 that supports components (includingproperties and events), objects, inheritance, polymorphism, reflection,and provides an object-oriented component-based application programminginterface (API), such as that of the .NET™ Framework manufactured byMicrosoft Corporation, Redmond, Wash. The device 1500 is of a very basicconfiguration demarcated by a dashed line 1508. Again, a terminal mayhave fewer components but will interact with a computing device that mayhave such a basic configuration.

Computing device 1500 may have additional features or functionality. Forexample, computing device 1500 may also include additional data storagedevices (removable and/or non-removable) such as, for example, magneticdisks, optical disks, or tape. Such additional storage is illustrated inFIG. 15 by removable storage 1509 and non-removable storage 1510.Computer storage media may include volatile and nonvolatile, removableand non-removable media implemented in any method or technology forstorage of information, such as computer readable instructions, datastructures, program modules, or other data. System memory 1504,removable storage 1509 and non-removable storage 1510 are all examplesof computer storage media. Computer storage media includes, but is notlimited to, RAM, ROM, EEPROM, flash memory or other memory technology,CD-ROM, digital versatile disks (DVD) or other optical storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to store thedesired information and which can be accessed by computing device 1500.Any such computer storage media may be part of device 1500. Computingdevice 1500 may also have input device(s) 1512 such as keyboard, mouse,pen, voice input device, touch input device, etc. Output device(s) 1514such as a display, speakers, printer, etc. may also be included. Thesedevices are well know in the art and need not be discussed at lengthhere.

Computing device 1500 may also contain communication connections 1515that allow the device to communicate with other computing devices 1518,such as over a network (e.g., consider the aforementioned network 410 ofFIG. 4). Communication connections 1515 are one example of communicationmedia. Communication media may typically be embodied by computerreadable instructions, data structures, program modules, etc.

CONCLUSION

Although the invention has been described in language specific tostructural features and/or methodological acts for prioritizing qualityimprovements to source code, it is to be understood that the inventionis not necessarily limited to the specific features or acts described.Rather, the specific features and acts are disclosed as exemplary formsof implementing the invention.

What is claimed is:
 1. A method, implemented at least in part by acomputing device, the method comprising: receiving source code thatcomprises a plurality of code segments; providing threshold levels ofcode coverage for the plurality of code segments, wherein the thresholdlevels of code coverage comprise a level as a function of complexity ofa respective code segment and a desired level of quality for the sourcecode, in which the complexity of the respective code segment determinesthe threshold level of code coverage for the desired level of qualitysuch that different threshold levels of code coverage are assigned tocode segments of different complexity to achieve a same desired level ofquality for the source code; providing results from test cases executedon the source code; identifying a code segment that does not meet itsthreshold level of code coverage; and recommending one or moreadditional test cases to increase a level of code coverage for theidentified code segment, wherein the one or more additional test casesare recommended based at least in part on one or more complexitymeasures and the desired level of quality for the source code.
 2. Themethod of claim 1 further comprising displaying a name for the codesegment that does not meet its threshold level of code coverage, acomplexity metric for the code segment and a current level of codecoverage.
 3. The method of claim 2 further comprising displaying a risklevel for the code segment where the risk level corresponds to a risk offailure of the code segment during execution of the code segment.
 4. Themethod of claim 3 further comprising calculating a quality assessmentvalue for a code segment based on a complexity measure for the codesegment divided by a level of code coverage for the code segment.
 5. Themethod of claim 2 further comprising executing the one or moreadditional test cases.
 6. The method of claim 5 further comprisingdetermining whether the code segment meets its threshold level of codecoverage upon execution of the one or more additional test cases.
 7. Acomputing device comprising: an interface to receive source code thatcomprises a plurality of code segments; a processor; a qualityassessment module executable by the processor to determine a desiredlevel of quality for the source code, wherein the quality assessmentmodule defines a quality assessment value based at least in part on acomplexity measure and a level of code coverage; a complexity moduleexecutable by the processor to analyze the source code to assign acomplexity measure to a code segment of the plurality of code segments,the plurality of code segments comprising a first code segment and asecond code segment; and a code coverage module executable by theprocessor to assign a level of code coverage to the code segment as afunction of the desired level of quality and the complexity measure, inwhich the complexity measure of the code segment determines the level ofcode coverage for the code segment for the desired level of quality,wherein a higher level of code coverage is provided for the first codesegment having a higher structural complexity than the second codesegment having a lower structural complexity to achieve a same desiredlevel of quality.
 8. The computing device of claim 7 wherein the codecoverage module is associated with a relationship between code quality,code complexity and code coverage.
 9. The computing device of claim 8wherein the relationship comprises a relationship based at least in parton a prior version of the source code.
 10. The computing device of claim8 wherein the relationship comprises a relationship based at least inpart on results from test cases written for the source code.
 11. Thecomputing device of claim 8 wherein the relationship comprises arelationship based at least in part on results from test cases executedon the source code.
 12. The computing device of claim 7 wherein thesource code comprises code written in an object-oriented programminglanguage.
 13. The computing device of claim 7 wherein the source codecomprises code written in an intermediate language.
 14. The computingdevice of claim 7 wherein the source code comprises code written in amark-up language.
 15. A storage memory device including instructionsexecutable by a processor that, when executed, performs operationscomprising: receiving a list of code segments for source code of anapplication; receiving a threshold level of code coverage for one ormore of the plurality of code segments, wherein the threshold level ofcode coverage comprises a level as a function of complexity of arespective code segment and a desired level of quality for the sourcecode, in which the complexity of the respective code segment determinesthe level for the desired level of quality for the source code, whereina higher threshold level of code coverage is provided for a code segmentmore complex than another code segment to achieve the desired level ofquality for the source code; receiving results from test cases executedon the source code; identifying code segments in the list that do notmeet their respective threshold levels; recommending additional testcases to increase level of code coverage for the identified codesegments; and prioritizing the additional test cases.
 16. The storagememory device of claim 15 wherein the prioritizing comprisesprioritizing the additional test cases based at least in part oncomplexity.
 17. The storage memory device of claim 15 wherein theprioritizing comprises prioritizing the additional test cases based atleast in part on threshold levels.
 18. The storage memory device ofclaim 15 wherein the prioritizing comprises prioritizing the additionaltest cases based at least in part on a risk of failure for the one ormore of the code segments.
 19. The storage memory device of claim 15further comprising calculating a quality assessment value for a codesegment in the list based at least in part on a complexity measure forthe code segment divided by a level of code coverage for the codesegment.
 20. The storage memory device of claim 15, further comprisingcalculating a quality assessment value for a code segment in the listbased at least in part on a complexity measure for the code segment anda level of code coverage.